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Bid  Protests  on  DoD  Source  Selections 


Douglas  J.  Buettner  and  Philip  S.  Anton 
June  13,  2017 


Summary 

The  Office  of  the  Secretary  of  Defense  (OSD)  Acquisition  Technology  &  Logistics  (AT&L) 
closely  monitors  the  Government  Accountability  Office  (GAO)  Bid  Protest  statistics  for  trends. 
These  statistics  combined  with  our  trend  analysis  provide  an  important  independent  assessment  of 
how  well  our  source  selection  process  is  working.  When  counting  sustainment  rates  against  all 
DoD  solicitations  (not  just  the  number  of  protests),  we  find  the  rate  to  be  about  0.02  percent. 

This  white  paper  provides  an  update  that  includes  the  Fiscal  Year  2016  trends  and  adds  our  new 
analysis  of  the  GAO’s  binning  of  contract  values.  Unlike  our  prior  analysis  featured  in  the 
USD/AT&L’s  Performance  of  the  Defense  Acquisition  System  reports  [Kendall  2016],  this 
analysis  only  uses  data  provided  directly  by  the  GAO,  and  doesn’t  utilize  bid  protest  statistics  from 
other  sources.  We  also  fixed  inconsistencies  that  we  identified  in  the  book  keeping  between  filed 
and  closed  years  and  made  an  attempt  to  identify  which  GAO  data  field  is  being  plotted,  and 
included  Figure  6  that  plots  the  sustained  protests  by  fiscal  year  filed  and  closed. 

To  summarize  the  contents  below,  we  found  that  GAO  sustainment  rates  per  solicitation  over  the 
number  of  bid  protests  for  each  fiscal  year  (closed)1  remained  statistically  flat  in  both  number  and 
dollar  value,  with  an  increasing  trend  in  protest  rates  (both  in  absolute  numbers  before  2010  and 
per  budget  dollar  over  recent  years).  Combined,  these  indicators  do  not  appear  to  suggest  a  decline 
in  the  quality  of  DoD  solicitations.  Instead,  the  increasing  protest  rate  per  budget  dollar  seems  to 
indicate  an  increased  propensity  by  bidders  to  protest.  GAO  has  sustained  a  few  protests  in  very- 
large  dollar  value  procurements,  but  these  sustainments  are  infrequent  and  a  recent  increase  in  the 
last  fiscal  year  is  not  statistically  significant.  Small  numbers  of  very-large-value  protests  tend  to 
drive  the  dollars  under  protest  and  total  value  of  sustained  protests. 


1  Our  use  of  "GAO  sustainment  rate"  or  "sustainment  rate"  is  the  number  of  sustained  protests  per  solicitation  over 
the  number  of  bid  protests  for  each  fiscal  year  (closed).  This  is  a  different  rate  than  the  GAO's  "sustainment  rate" 
which  is  the  number  of  sustained  protests  per  the  number  of  merit  cases  over  the  number  of  bid  protests  for  each 
fiscal  year  (closed).  Our  number  is  reflective  of  a  GAO  sustainment  decision  against  our  DoD  bid  solicitations. 
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Introduction 


A  bid  protest  is  a  legal  challenge  to  a  procurement  action  (a  solicitation  or  associated  award)  for 
the  procurement  of  goods  or  services.2  The  GAO  is  a  primary  adjudicator  of  bid  protests 
concerning  Federal  agencies  and  departments  (including  the  DoD).3  Once  a  protest  is  filed,  the 
GAO  has  100  days  to  issue  a  decision  (31  U.S.C.,  Sec.  3554)  or  if  the  GAO  decides  to  proceed 
under  its  express  option  procedures  (under  4  CFR  21.9)  it  will  render  a  decision  in  65  days. 

A  protest  can  have  one  of  four  outcomes:  it  can  be  withdrawn  by  the  protester  at  any  time, 
dismissed  by  the  GAO  in  its  initial  review,  or  denied  or  sustained  by  the  GAO  in  its  final  decision.4 
Oftentimes,  withdrawals  and  dismissals  are  a  direct  result  of  agency-initiated  corrective  action.  In 
such  an  event,  no  clear  understanding  of  the  merits  can  be  derived  from  the  metrics.  Final  decisions 
(denials  and  sustainments)  clearly  indicate  the  merits  of  each  case  and  are  supported  by  separate 
decision  documents.  GAO’s  data  tracks  the  outcome  for  each  protest,  the  name  of  the  protesting 
organization,  the  solicitation  number,  and  the  dollar  range  of  the  solicitation’s  value. 

Number  and  Value  of  Protests 

Figure  1  shows  the  total  number  of  protest  cases  received  by  the  GAO  for  DoD  and  Federal  non¬ 
defense  actions  by  filing  year.  5  The  number  of  bid  protests  against  DoD  actions  generally  rose  in 
mid-2000s,  then  leveled  off  at  about  1,300  per  year  since  FY  2010.  The  number  of  protests  in  FY 
2016  dropped  to  the  lowest  level  since  FY  2009,  but  this  followed  a  peak  increase  in  FY  2015. 
The  trend  in  non-defense  protests  generally  mirrored  defense  protests  but  with  a  1-2  year  lag.  As 
we  saw  in  last  year’s  report  (USD[AT&L],  2016b),  the  number  of  protests  by  Component  (or 
military  service  is  not  shown)  has  been  relatively  consistent  since  2009,  with  somewhat  of  a  drop 
for  the  Army  in  the  last  two  years. 


2  Challenges  include  protests,  cost  claims,  and  requests  for  reconsideration.  This  report  discusses  protests  only.  The 
GAO  reports  case  types  by  docket  numbers  (B-#)  where  the  docket  number  aligns  to  a  Federal  or  DoD  Request  for 
Proposal  (RFP)  or  Request  for  Quote  (RFQ)  solicitation  number  and  includes  challenges  other  than  "protests"  (i.e., 
requests  for  reconsideration,  entitlement,  and  claims  for  costs). 

3  Bid  protests  can  also  be  handled  by  the  procuring  agency  or  the  U.S.  Court  of  Federal  Claims.  Thus,  these  data 
provide  a  view  into  a  significant  number  of  (but  not  all)  bid  protests. 

4  Note  that  bid  protests  are  an  integral  part  of  the  government  acquisition  system  to  help  ensure  that  all  bidders  are 
treated  fairly  and  that  solicitations  and  awards  are  conducted  properly.  There  is  no  analogous  bid  protest  process 
for  commercial  business  transactions.  Despite  some  limitations,  the  number  of  protests  sustained  by  the  GAO 
(independent  of  the  number  of  protests)  provides  an  indicator  of  the  quality  of  the  DoD's  source  selections.  See 
http://www.gao.gov/legal/bids/bidprotest.html  as  well  as  Schwartz  and  Manuel  (2015)  for  further  information  on 
the  GAO  process  and  timeline. 

5ln  this  annual  report  we  distinguish  between  the  years  in  which  a  protest  was  filed  from  the  year  in  which  a  protest 
was  closed;  often  these  are  the  same— but  not  always.  To  accomplish  this,  we  restrict  our  analysis  to  data  received 
directly  from  the  GAO  as  well  as  GAO's  short  annual  reports  to  Congress.  As  a  result,  many  of  our  plots  only  go  back 
to  FY  2006  in  this  annual  report  compared  to  prior  reports,  which  included  data  from  Schwartz  and  Manuel  (2015); 
those  data  do  not  distinguish  between  filing  and  closure  years. 
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When  plotting  the  number  of  protests  per  DoD  budget  dollar  (excluding  military  personnel),  we 
find  the  protest  rate  per  dollar  has  been  rising  steadily  (see  Figure  2).  This  rise  is  seen  throughout 
these  data  and  did  not  begin  when  DoD  budgets  began  to  decline  in  FY  2008.  Using  GAO’s  data 
on  the  dollar- value  range  for  each  protest,6  we  bound  the  minimum  and  maximum  total  value  of 
the  actions  under  protest  for  each  year.  Most  protested  actions  were  valued  well  below  half  a  billion 
dollars;  about  a  third  were  worth  less  than  $10  million  and  another  third  had  unknown  values  (see 
Figure  3). 

A  few  protests  against  very  large  procurements  seem  to  be  driving  the  increasing  value  of  actions 
under  protest.  Figure  4  shows  an  increasing  trend  in  the  values  of  protested  solicitations  since  FY 
2006.  If  the  three  extremely  large  protests  are  removed,  there  was  still  a  slight  trend  since  FY  2006 
but  it  disappears  if  measured  since  2009.  Also,  if  we  remove  the  23  large  protests  greater  than  $3 
billion,  then  there  is  no  clear  trend  because  of  the  uncertainty  bounds  in  GAO’s  value  data  (see 
Figure  5). 

Figure  1.  Number  of  Protest  Received  by  GAO:  DoD  and  Federal  Non-DoD  (FY  2006- 
2016) 


#  of  Bid  Protests 


2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016 

Fiscal  Year  (filed) 

SOURCE:  GAO 

NOTE:  The  number  of  civilian  cases  in  a  fiscal  year  is  derived  from  the  GAO’s  Annual  Reports  and  the  GAO’s  provided  database  of  DoD  filings. 
Unlike  versions  in  prior  annual  reports,  this  plot  only  shows  data  going  back  to  FY  2006  when  we  can  verify  the  counts  in  filing  year. 


6  The  GAO  only  provided  values  in  ranges  (e.g.,  greater  than  $500  million  but  less  than  $1B)  or  as  an  indicator  ( micro¬ 
purchase  or  unknown)  for  each  solicitation.  For  protests  over  $1  billion,  we  researched  the  value  from  FPDS-NG  or 
FedBizOpps  using  the  solicitation  number.  Micro-purchases  were  valued  up  to  $3,000  from  FY  2006  to  July  of  FY 
2015;  afterwards  it  was  increased  to  $3,500  (see  FAR  Subpart  13.2).  The  threshold  is  now,  as  result  of  legislation  in 
FY  17  NDAA  $5,000,  but  this  increase  took  place  too  late  to  affect  the  analysis  here.  We  did  not  research  the  values 
of  the  large  number  (3,079)  of  unknown-valued  protests,  but  only  20  were  sustained  and  244  were  denied. 
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Figure  2.  Protests  per  DoD  Budget  Dollars  (FY  2006-2016) 


#  Protests  per  FY17$,  billion 


2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016 

Fiscal  Year  (filed) 

Source:  GAO;  USD(C),  2016c  (Green  Book,  Table  6-1). 

NOTE:  Annual  budgets  are  DoD  TOAs  in  constant  FY  17  dollars,  excluding  military  personnel  budgets.  The  upward  trend  is  marked  with  a  straight 
red  line  (R2=  84%). 


Figure  3.  Protested  Solicitation  Values  (FY  2006-2016) 


Number  of  Protests 


purchase  $100K  $5M  $10M  $50M  $100M  $500M  $1B 

Source:  GAO 


NOTE:  N=8,196  protests.  Micro-purchases  were  valued  up  to  $3,000  through  July  of  FY  2015;  afterwards  it  was  $3,500  (see  FAR  Subpart  13.2). 
GAO’s  bin  values  are  not  of  equal  or  increasing  ranges,  which  affects  the  profile. 
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Figure  4.  All  GAO  Protested  Contract  Solicitation  FY17$  Value  Totals  per  Fiscal  Year  (FY 
2006-2016) 


FY17$,  Billions 


2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016 

Fiscal  Year  (filed)  x  Protests  valued  at  >  $30B 

Sources:  GAO,  FedBizOpps,  FPDS-NG,  and  other  sources. 

NOTE:  The  three  protests  valued  at  over  $30  billion  are  plotted  and  included  in  the  shaded  band. 

Figure  5.  All  GAO  Protested  Contract  Solicitation  Value  Totals  per  Fiscal  Year  Removing 
Protests  with  Values  greater  or  equal  to  $3  billion  and  Inflation  (FY  2006-2016) 

FY17$,  Billions 


Fiscal  Year  (filed) 

Sources:  GAO,  FedBizOpps,  FPDS-NG,  Green  Book  Table  5-4,  and  other  sources. 
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Sustained  Protests 


Despite  this  increasing  protest  rate  per  dollar,  the  sustainment  numbers  and  rates  have  remained 
relatively  flat  but  for  apparent  outliers.  There  was  an  increase  in  sustainment  protests  closed  in  FY 
2016,  but  when  plotted  by  filing  year7  the  recent  pattern  changed  from  an  increase  to  a  decrease 
(see  Figure  6).  There  is  no  statistically  significant  trend  since  FY  2007  in  either  measure.  When 
controlling  for  multiple  protests  on  the  same  solicitation  (not  shown),  the  numbers  drop  from  about 
30  sustainments  per  year  to  about  15,  but  the  increase  in  FY  2016  closures  remain. 

Sustainment  rates  have  been  declining  somewhat  and  have  been  running  at  about  2  percent,  except 
for  the  increase  in  FY  2016  (see  Figure  7).  These  trends  remain  when  removing  multiple 
sustainments  on  the  same  solicitation  (see  the  green  dashed  Protested  Solicitations  line  on  the 
plot).  When  counting  sustainment  rates  against  all  DoD  solicitations  (not  just  the  number  of 
protests),  we  find  similar  trends  but  at  about  a  hundredth  the  rate  (about  0.02  percent;  see  the  purple 
dashed  All  Solicitations  line  on  the  plot). 

When  measuring  sustainments  in  dollar  values,  we  again  see  that  a  few  very-large  sustainments 
dominate,  especially  the  KC-X  solicitation  (now  KC-46A  program),  two  troop-support  contracts 
(one  for  Kuwait,  Iraq,  and  Jordan,  and  another  for  Afghanistan),  and  the  Encore  III  information 
technology  services  contract  (see  Figure  8).  When  removing  apparent  outliers  valued  at  over  $3 
billion  (FY  17  dollars),  we  find  that  three  sustainments8  in  FY  2015-2016  drove  the  recent  upswing 
(see  Figure  9). 

Besides  definitive  sustainments,  it  would  be  useful  to  know  the  extent  to  which  the  DoD  provides 
any  accommodations  or  corrective  actions  (warranted  and  unwarranted)  to  help  identify  trends  in 
the  department’s  performance.  Unfortunately,  there  is  no  central  data  collection  within  the  DoD 
on  such  actions.  The  GAO  does  report  Federal-wide  rates  of  actions  (regardless  of  whether  they 
are  warranted  or  not)  in  its  annual  reports  on  bid  protests  to  Congress,  but  GAO  does  not  track 
these  actions  by  solicitation  nor  the  overall  rate  for  just  the  DoD.  Still,  if  we  assume  that  the  DoD’s 
"effectiveness"  rate  was  the  same  as  that  reported  for  the  entire  Federal  government  (currently 
46%  of  GAO  filed  protests  in  FY  2016),  then  about  1%  of  solicitations  involved  protests  in  which 
the  agency  made  some  kind  of  corrective  action  or  accommodation  to  the  bidder.  We  do  not  know 
what  portion  of  these  actions  were  warranted  or  the  valuation  profile  for  these  actions  as  this  data 
is  not  collected,  but  this  extrapolation  indicates  that  only  a  very  small  percentage  of  DoD 
solicitations  may  involve  corrective  actions  (warranted  or  not). 


7  There  is  a  differences  between  plotting  by  protest  year  and  closure  year  since  the  GAO  may  receive  protests  in  one 
fiscal  year  that  are  resolved  in  the  following  year. 

8  Solicitations  for  the  Navy's  Consolidated  Afloat  Networks  and  Enterprise  Services  (CANES),  the  Air  Force's  Arnold 
Engineering  Development  Complex's  (AEDC)  Test  O&S,  and  the  DoD  Freight  Transportation  Service  (DFTS). 
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Figure  6.  Number  of  Protested  DoD  Solicitations  Sustained  by  GAO  (FY  2006-2016) 


#  Solicitations  Sustained 


SOURCE:  GAO 


Fiscal  Year 


NOTE:  These  two  lines  are  statistically  flat  (i.e.,  with  an  R2  of  only  1-3%  the  trend  lines  would  explain  practically  none  of  the  variation  seen).  The 
FY  2016  value  by  filed  date  is  incomplete.9 


9  Our  GAO  provided  bid  protest  data  only  contains  cases  that  were  closed  on  or  before  the  completion  of  FY  2016. 
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Figure  7.  GAO  Protest  and  Solicitation  Sustainment  Rates  (FY  2006-2016) 


Sustainment  Rates 


2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016 

Fiscal  Year  (closed) 

Sources:  GAO,  FEDBizOps 

NOTES:  The  Protests  rate  is  the  sustainment  rate  against  the  number  of  protests  closed  in  the  Fiscal  Year.  The  Protested  Solicitations  rate  is  number 
of  closed  solicitations  with  at  least  one  sustained  protest  against  the  number  of  protested  solicitations.  The  lOOx  (All  FY  Solicitations )  rate  is  the 
number  of  sustainments  (times  100  in  order  to  plot  it  on  the  same  scale  as  the  other  rates)  against  the  number  of  all  solicitations  (protested  or  not). 
The  FY  2006  values  for  the  last  two  measures  may  be  affected  by  initial  data  collection,  so  those  apparent  drops  may  not  be  reliable.  In  FY  2009 
the  number  of  sustained  protests  was  less  than  the  number  of  sustained  solicitations  (16  versus  18),  but  in  percentage  the  rate  of  protested 
solicitations  was  higher  because  there  were  many  more  protest  cases  than  protested  solicitations. 
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Figure  8.  Total  Value  of  Sustained  Protests  (FY  2006-2016) 


FY17$,  Billions 


2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016 

Fiscal  Year  (closed)  x  sustained  protests  >  $1B 

Source:  GAO 

NOTE:  The  sustained  protests  that  drove  the  spikes  are  show  with  X’s;  they  include  six  sustained  protests  valued  at  over  $3  billion  and  three  recent 
protests  valued  at  over  $1.5  billion,  all  of  which  are  included  in  the  shaded  band. 

Figure  9.  Total  Value  of  Sustained  Protests  Worth  Less  Than  $3  Billion  (FY  2006-2016) 
FY17$,  Billions 


Source:  GAO 

NOTE:  The  three  recent  sustained  protests  valued  at  over  $1.5  billion  are  plotted  and  included  in  the  shaded  band. 
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